An Average-Reward Reinforcement Learning Algorithm for Computing Bias-Optimal Policies

Computing Bias-Optimal Policies Sridhar Mahadevan Department of Computer Science and Engineering University of South Florida Tampa, Florida 33620 [email protected] Abstract Average-reward reinforcement learning (ARL) is an undiscounted optimality framework that is generally applicable to a broad range of control tasks. ARL computes gain-optimal control policies that maximize the expected payo per step. However, gainoptimality has some intrinsic limitations as an optimality criterion, since for example, it cannot distinguish between di erent policies that all reach an absorbing goal state, but incur varying costs. A more selective criterion is bias optimality, which can lter gain-optimal policies to select those that reach absorbing goals with the minimum cost. While several ARL algorithms for computing gain-optimal policies have been proposed, none of these algorithms can guarantee bias optimality, since this requires solving at least two nested optimality equations. In this paper, we describe a novel model-based ARL algorithm for computing bias-optimal policies. We test the proposed algorithm using an admission control queuing system, and show that it is able to utilize the queue much more e ciently than a gain-optimal method by learning bias-optimal policies. Motivation Recently, there has been growing interest in an undiscounted optimality framework called average reward reinforcement learning (ARL) (Boutilier & Puterman 1995; Mahadevan 1994; 1996a; Schwartz 1993; Singh 1994; Tadepalli & Ok 1994). ARL is wellsuited to many cyclical control tasks, such as a robot avoiding obstacles (Mahadevan 1996a), an automated guided vehicle (AGV) transporting parts (Tadepalli & Ok 1994), and for process-oriented planning tasks (Boutilier & Puterman 1995), since the average reward is a good metric to evaluate performance in these tasks. However, one problem with the average reward criterion is that it is not su ciently selective, both in goal-based tasks and tasks with no absorbing goals. Figure 1 illustrates the limitation of the average reward criterion on a simple two-dimensional grid-world task. Here, the learner is continually rewarded by +10 for reaching and staying in the absorbing goal stateG, and is rewarded 1 in all non-goal states. Clearly, all control policies that reach the goal will have the same average reward. Thus, the average reward criterion cannot be used to select policies that reach absorbing goals in the shortest time.